Method for Classifying Cells

ABSTRACT

The disclosure provides example embodiments for automatically or semi-automatically classifying cells in microscopic images of biological samples. These embodiments include methods for selecting training sets for the development of classifier models. The disclosed selection embodiments can allow for the re-training of classifier models using training examples that have been subjected to the same or similar incubation conditions as target samples. These selection embodiments can reduce the amount of human effort required to specify the training examples. The disclosed embodiments also include the classification of individual cells based on metrics determined for the cells using phase contrast imagery and defocused brightfield imagery. These metrics can include size, shape, texture, and intensity-based metrics. These metrics are determined based on segmentation of the underlying imagery. The segmentation is based, in some embodiments, on phase contrast imagery and/or defocused brightfield imagery of biological samples.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation-in-part of U.S. application Ser. No.16/265,910, filed Feb. 1, 2019, the contents of which are herebyincorporated by reference. The contents of U.S. application Ser. No.______, docket no. 18-2075-US-CIP, filed on Nov. 17, 2020, entitled“Computational Model for Analyzing Images of a Biological Specimen” isalso hereby incorporated by reference.

BACKGROUND

Current known methods of segmenting cells in biological specimensrequire fluorescently-labeled proteins, e.g., thresholding anuclear-localizing protein like histones for a marker-controlledsegmentation algorithm. Alternative label-free techniques exist, such asptychography-based methods, lateral shearing interferometry and digitalholography, but these require a complex image acquisition setup and acomplicated image formation algorithm with long processing times.Another label-free technique includes deep-learning algorithms (e.g.,convolutional neural networks) that require extensive training on largedata sets of images and slow processing times. Other methods use abrightfield image in an out-of-focus condition that requires specializedhardware like a pinhole aperture and do not permit cell-by-cellsegmentation.

Classification of cells in microscopy images (e.g., of cells whoselocation and extent within the image have been determined bysegmentation) can facilitate a variety of applications, includingassessment of the effects of a variety of experimental conditions byquantifying the effects of those conditions in terms of the increase ordecrease in the number of cells present in a sample and/or a proportionof the cells that correspond to a variety of conditions (e.g.,differentiated vs. non-differentiated). Cell classification can beperformed manually, however, such manual classification can be expensivein terms of time and effort and may result in inaccurate classificationof cells. Automated methods are also available, however, these methodsmay require fluorescently-labeled proteins, which can interrupt thenatural biology of the cells, or may require providing large sets oftraining examples to train the automated algorithms.

SUMMARY

In one aspect, an example method for classification of cells isdisclosed. The method includes: (i) obtaining a set of images of aplurality of biological samples, wherein the set of images includes atleast one image of each sample of the plurality of biological samples;(ii) obtaining an indication of a first set of cells within theplurality of biological samples and obtaining an indication of a secondset of cells within the plurality of biological samples, wherein thefirst set of cells is associated with a first condition and the secondset of cells is associated with a second condition; (iii) based on theset of images, the indication of the first set of cells, and theindication of the second set of cells, determining a first plurality ofsets of metrics, wherein the first plurality of sets of metrics includea set of metrics for each cell of the first set of cells and a set ofmetrics for each cell of the second set of cells; (iv) based on thefirst plurality of sets of metrics, using a supervised learningalgorithm to generate a model to distinguish between cells in the firstset of cells and cells in the second set of cells, thereby generating atrained model; (v) based on the set of images, determining a secondplurality of sets of metrics, wherein the second plurality of sets ofmetrics include a set of metrics for each cell present in a targetsample; and (vi) classifying a cell in the target sample, whereinclassifying the cell includes applying the trained model to the set ofmetrics for the cell.

In another aspect, an example method for classification of cells isprovided. The method includes: (i) obtaining three or more images of atarget sample, wherein the target sample includes one or more cellscentered around a focal plane for the target sample, wherein the threeor more images include a phase contrast image, a first brightfieldimage, and a second brightfield image, wherein the first brightfieldimage represents an image of the target sample focused at a firstdefocusing distance above the focal plane, and wherein the secondbrightfield image represents an image of the target sample focused at asecond defocusing distance below the focal plane; (ii) determining acell image of the target sample based on the first and secondbrightfield images; (iii) determining a target segmentation map for thetarget sample based on the cell image and the phase contrast image; (iv)based on the two or more images of the target sample and the targetsegmentation map, determining a set of metrics for each cell present inthe target sample; and (v) classifying a cell in the target sample,wherein classifying the cell includes applying the set of metrics of thecell to a trained classifier.

In yet another aspect, an example method for classification of cells isprovided. The method includes: (i) obtaining two or more images of atarget sample, wherein the target sample includes one or more cellscentered around a focal plane for the target sample, wherein the two ormore images include a phase contrast image and one or more brightfieldimages, wherein the one or more brightfield images includes at least onebrightfield image that represents an image of the target sample that isnot focused at the focal plane; (ii) based on the two or more images,determining a set of metrics for each cell present in the target sample;and (iii) classifying a cell in the target sample by applying a trainedmodel to the set of metrics for the cell.

In another aspect, a non-transitory computer-readable medium is providedthat is configured to store at least computer-readable instructionsthat, when executed by one or more processors of a computing device,cause the computing device to perform controller operations to performany of the above methods.

In yet another aspect, a system for assaying biological specimens isprovided that includes: (i) an optical microscope; (ii) a controller,wherein the controller comprises one or more processors; and (iii) anon-transitory computer-readable medium that is configured to store atleast computer-readable instructions that, when executed by thecontroller, cause the controller to perform controller operations toperform any of the above methods.

The features, functions, and advantages that have been discussed can beachieved independently in various examples or may be combined in yetother examples further details of which can be seen with reference tothe following description and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 is a functional block diagram of a system, according to oneexample implementation;

FIG. 2 depicts a block diagram of a computing device and a computernetwork, according to an example implementation;

FIG. 3 shows a flowchart of a method, according to an exampleimplementation;

FIG. 4 shows images of a biological specimen, according to an exampleimplementation;

FIG. 5 shows images of another biological specimen, according to anexample implementation;

FIG. 6A shows experimental results of a cell-by-cell segmentation mask,generated according to an example implementation, for a cell imageresponse at 24 hours after a time course of HT1080 fibrosarcomaapoptosis following a camptothecin (CPT, cytotoxic) treatment;

FIG. 6B shows cell subsets classified based on red (Nuclight Red, a cellhealth indicator, “NucRed”) and green fluorescence (Caspase 3/7, anapoptosis indicator), according to the implementation of FIG. 6A;

FIG. 6C shows there was a decrease in the red population after CPTtreatment indicating loss of viable cells, increasing red and greenfluorescence indicating early apoptosis, as well as increasing greenfluorescence after 24 hours indicating late apoptosis, according to theimplementation of FIG. 6A;

FIG. 6D shows concentration response time courses of the early apoptoticpopulation (percentage of total cells exhibiting red and greenfluorescence), according to the implementation of FIG. 6A;

FIG. 6E shows experimental results of a cell-by-cell segmentation mask,generated according to an example implementation, for a cell imageresponse at 24 hours after a time course of HT1080 fibrosarcomaapoptosis following a cyclohexamide (CHX, cytostatic) treatment;

FIG. 6F shows cell subsets classified based on red (Nuclight Red, a cellhealth indicator, “NucRed”) and green fluorescence (Caspase 3/7, anapoptosis indicator), according to the implementation of FIG. 6E;

FIG. 6G shows there was a lack of apoptosis but a decrease in cellcounts after CHX treatment, according to the implementation of FIG. 6E;

FIG. 6H shows concentration response time courses of the early apoptoticpopulation (percentage of total cells exhibiting red and greenfluorescence), according to the implementation of FIG. 6E;

FIG. 7A shows a cell-by-cell segmentation mask imposed over a phasecontrast image for label-free cell counting of adherent cells usingcell-by-cell segmentation analysis, generated according to an exampleimplementation. Various densities of A549 Cells labelled with NucLightRed reagent were analyzed with both the label-free cell-by-cell analysisand with the red nuclear count analysis to validate the label-freecounting over time;

FIG. 7B shows the cell-by-cell segmentation mask according to FIG. 7Awithout the phase contrast image in the background;

FIG. 7C shows a time course of phase count and NucRed count data acrossdensities, according to the implementation of FIG. 7A;

FIG. 7D shows a correlation of count data over 48 hours and demonstratesR2 value of 1 with a slope of 1, according to the implementation of FIG.7A;

FIG. 8 shows a flowchart of a method, according to an exampleimplementation;

FIG. 9 shows a flowchart of a method, according to an exampleimplementation;

FIG. 10 shows a flowchart of a method, according to an exampleimplementation;

FIG. 11 shows an example microscopic image and a related examplesegmentation map;

FIG. 12A shows an example annotated microscopic image;

FIG. 12B shows an example annotated microscopic image;

FIG. 13 shows an example schematic representation of wells of amulti-well sample plate;

FIGS. 14A and 14B illustrate the experimental predictive accuracy ofmethods described herein;

FIGS. 15A and 15B illustrate the experimental predictive accuracy ofmethods described herein; and

FIGS. 16A, 16B, and 16C illustrate the experimental predictive accuracyof methods described herein as compared to label-based classification.

The drawings are for the purpose of illustrating examples, but it isunderstood that the inventions are not limited to the arrangements andinstrumentalities shown in the drawings.

DETAILED DESCRIPTION

I. Overview

Microscopic imaging of biological samples can facilitate many analysesof the contents of the samples and of their responses to a variety ofapplied experimental conditions. Such analyses can include counting thecells after classifying the cells in order to determine the effect ofthe applied conditions. For example, a sample could include a set ofdifferentiated cells and a set of undifferentiated cells, and analysisof the sample could include determining the proportion of the cells thatare differentiated, e.g., in order to determine the effectiveness of anapplied condition in causing the undifferentiated cells to becomedifferentiated. To perform such an analysis it is necessary to localizeeach of the cells in the sample and then to classify each of the cells.Such a classification process could be performed manually. However,manual classification can be very expensive, time consuming, and canresult in inaccurate classifications.

Embodiments described herein proved a variety of methods forautomatically classifying cells based on phase contrast images,brightfield images, composites of phase contrast and/or brightfieldimages, or other microscopic imagery of the cells. Some of theseembodiments include using specified sets of cells within one or morebiological samples to train a model to classify the cells. Such atrained model can then be applied to additional cells to classify thoseadditional cells. In order to classify a particular cell, a set ofmetrics is determined for the cell based on one or more images thatrepresent the cell. Such metrics can include metrics related to the sizeand/or shape of the cell. Such metrics may additionally or alternativelybe related to the texture or intensity of the cell as represented in oneor more phase contrast images, brightfield images, fluorescence images,or composite images. For example, one or more of the metrics could berelated to the texture of the cells (e.g., the variability and/or thestructure of variability of brightness or intensity across the area ofthe cell) in fluorescence images or in some other variety of images(e.g., phase contrast, brightfield). The determined set of metrics for acell can then be applied to a trained model in order to classify thecells.

The sets of cells used to train the model can be identified in a varietyof ways. In some examples, the cells could be manually indicated by auser. This could include the user manually indicating whole wells of amulti-well sample plate. Additionally or alternatively, the user couldmanually indicate individual cells within one or more biologicalsamples. In yet another example, the user could specify points in timeto indicate sets of cells, e.g., setting a first point in time beforewhich all cells in a sample belong to a first set (e.g., anundifferentiated set) and setting a second point in time after which allcells in a sample belong to a second set (e.g., a differentiated set).In some examples, the cells could be automatically or semi-automaticallyindicated. This could include identifying sets of cells based onfluorescence images of the cells (e.g., cells with supra-thresholdfluorescence signals could be assigned to a first group, while cellswith sub-threshold fluorescence signals could be assigned to a secondgroup). In another example, an unsupervised or semi-supervised learningalgorithm could cluster or otherwise aggregate the cells into sets thatcould then be used to train a classifier.

II. Example Architecture

FIG. 1 is a block diagram showing an operating environment 100 thatincludes or involves, for example, an optical microscope 105 and abiological specimen 110 having one or more cells. Methods 300, 800, 900,and 1000 in FIGS. 3-5, 8, 9, and 10 described below shows embodiments ofmethods that can be implemented within this operating environment 100.

FIG. 2 is a block diagram illustrating an example of a computing device200, according to an example implementation, that is configured tointerface with operating environment 100, either directly or indirectly.The computing device 200 may be used to perform functions of methodsshown in FIGS. 3-5, 8, 9, and 10 and described below. In particular,computing device 200 can be configured to perform one or more functions,including image generating functions that are based, in part, on imagesobtained by the optical microscope 105, for example. The computingdevice 200 has a processor(s) 202, and also a communication interface204, data storage 206, an output interface 208, and a display 210 eachconnected to a communication bus 212. The computing device 200 may alsoinclude hardware to enable communication within the computing device 200and between the computing device 200 and other devices (e.g. not shown).The hardware may include transmitters, receivers, and antennas, forexample.

The communication interface 204 may be a wireless interface and/or oneor more wired interfaces that allow for both short-range communicationand long-range communication to one or more networks 214 or to one ormore remote computing devices 216 (e.g., a tablet 216 a, a personalcomputer 216 b, a laptop computer 216 c and a mobile computing device216 d, for example). Such wireless interfaces may provide forcommunication under one or more wireless communication protocols, suchas Bluetooth, WiFi (e.g., an institute of electrical and electronicengineers (IEEE) 802.11 protocol), Long-Term Evolution (LTE), cellularcommunications, near-field communication (NFC), and/or other wirelesscommunication protocols. Such wired interfaces may include Ethernetinterface, a Universal Serial Bus (USB) interface, or similar interfaceto communicate via a wire, a twisted pair of wires, a coaxial cable, anoptical link, a fiber-optic link, or other physical connection to awired network. Thus, the communication interface 204 may be configuredto receive input data from one or more devices, and may also beconfigured to send output data to other devices.

The communication interface 204 may also include a user-input device,such as a keyboard, a keypad, a touch screen, a touch pad, a computermouse, a track ball and/or other similar devices, for example.

The data storage 206 may include or take the form of one or morecomputer-readable storage media that can be read or accessed by theprocessor(s) 202. The computer-readable storage media can includevolatile and/or non-volatile storage components, such as optical,magnetic, organic or other memory or disc storage, which can beintegrated in whole or in part with the processor(s) 202. The datastorage 206 is considered non-transitory computer readable media. Insome examples, the data storage 206 can be implemented using a singlephysical device (e.g., one optical, magnetic, organic or other memory ordisc storage unit), while in other examples, the data storage 206 can beimplemented using two or more physical devices.

The data storage 206 thus is a non-transitory computer readable storagemedium, and executable instructions 218 are stored thereon. Theinstructions 218 include computer executable code. When the instructions218 are executed by the processor(s) 202, the processor(s) 202 arecaused to perform functions. Such functions include, but are not limitedto, receiving brightfield images from the optical microscope 100 andgenerating a phase contrast image, a confluence mask, a cell image, aseed mask, a cell-by-cell segmentation mask and fluorescent images.

The processor(s) 202 may be a general-purpose processor or a specialpurpose processor (e.g., digital signal processors, application specificintegrated circuits, etc.). The processor(s) 202 may receive inputs fromthe communication interface 204, and process the inputs to generateoutputs that are stored in the data storage 206 and output to thedisplay 210. The processor(s) 202 can be configured to execute theexecutable instructions 218 (e.g., computer-readable programinstructions) that are stored in the data storage 206 and are executableto provide the functionality of the computing device 200 describedherein.

The output interface 208 outputs information to the display 210 or toother components as well. Thus, the output interface 208 may be similarto the communication interface 204 and can be a wireless interface(e.g., transmitter) or a wired interface as well. The output interface208 may send commands to one or more controllable devices, for example

The computing device 200 shown in FIG. 2 may also be representative of alocal computing device 200 a in operating environment 100, for example,in communication with optical microscope 105. This local computingdevice 200 a may perform one or more of the steps of the methods 300,800, 900, 1000 described below, may receive input from a user and/or maysend image data and user input to computing device 200 to perform all orsome of the steps of methods 300, 800, 900, and/or 1000. In addition, inone optional example embodiment, the Incucyte® platform may be utilizedto perform one or more of methods 300, 800, 900, 1000 and includes thecombined functionality of computing device 200 and optical microscope105.

FIG. 3 shows a flowchart of an example method 300 to achievecell-by-cell segmentation for one or more cells of a biological specimen110, according to an example implementation. FIGS. 8, 9, and 10 showflowcharts of an example methods 800, 900, and 1000, respectively, toachieve cell-by-cell classification of one or more cells of a biologicalspecimen 110, according to example implementations. Methods 300, 800,900, 1000 shown in FIGS. 3, 8, 9, 10 present examples of methods thatcould be used with the computing device 200 of FIG. 2, for example.Further, devices or systems may be used or configured to perform logicalfunctions presented in FIGS. 3, 8, 9, and/or 10. In some instances,components of the devices and/or systems may be configured to performthe functions such that the components are configured and structuredwith hardware and/or software to enable such performance. Components ofthe devices and/or systems may be arranged to be adapted to, capable of,or suited for performing the functions, such as when operated in aspecific manner. Methods 300, 800, 900, 100 may include one or moreoperations, functions, or actions as illustrated by one or more of theblocks in those figured (e.g., blocks 305-330). Although the blocks ofeach method are illustrated in a sequential order within each figure,some of these blocks may also be performed in parallel, and/or in adifferent order than those described herein. Also, the various blocksmay be combined into fewer blocks, divided into additional blocks,and/or removed based upon the desired implementation.

It should be understood that for this and other processes and methodsdisclosed herein, flowcharts show functionality and operation of onepossible implementation of the present examples. In this regard, eachblock may represent a module, a segment, or a portion of program code,which includes one or more instructions executable by a processor forimplementing specific logical functions or steps in the process. Theprogram code may be stored on any type of computer readable medium ordata storage, for example, such as a storage device including a disk orhard drive. Further, the program code can be encoded on acomputer-readable storage media in a machine-readable format, or onother non-transitory media or articles of manufacture. The computerreadable medium may include non-transitory computer readable medium ormemory, for example, such as computer-readable media that stores datafor short periods of time such as register memory, processor cache andRandom Access Memory (RAM). The computer readable medium may alsoinclude non-transitory media, such as secondary or persistent long termstorage, like read only memory (ROM), optical or magnetic disks,compact-disc read only memory (CD-ROM), for example. The computerreadable media may also be any other volatile or non-volatile storagesystems. The computer readable medium may be considered a tangiblecomputer readable storage medium, for example.

In addition, each block in FIGS. 3, 8, 9, 10, and within other processesand methods disclosed herein, may represent circuitry that is wired toperform the specific logical functions in the process. Alternativeimplementations are included within the scope of the examples of thepresent disclosure in which functions may be executed out of order fromthat shown or discussed, including substantially concurrent or inreverse order, depending on the functionality involved, as would beunderstood by those reasonably skilled in the art.

III. Example Methods

As used herein, a “brightfield image” refers to an image obtained via amicroscope based on a biological sample illuminated from below such thatthe light waves pass through transparent portions of the biologicalsample. The varying brightness levels are then captured in a brightfieldimage.

As used herein, a “phase contrast image” refers to an image obtained viaa microscope, either directly or indirectly, based on a biologicalsample illuminated from below capturing phase shifts of light passingthrough the biological sample due to differences in the refractive indexof different portions of the biological sample. For example, when lightwaves travel through the biological specimen, the light wave amplitude(i.e., brightness) and phase change in a manner dependent on propertiesof the biological specimen. As a result, a phase contrast image hasbrightness intensity values associated with pixels that vary such thatdenser regions with a high refractive index are rendered darker in theresulting image and thinner regions with a lower refractive index arerendered lighter in the resulting image. Phase contrast images can begenerated via a number of techniques, including from a Z-stack ofbrightfield images.

As used herein, a “Z-stack” or “Z-sweep” of brightfield images refers toa digital image processing method which combines multiple images takenat different focal distances to provide a composite image with a greaterdepth of field (i.e. the thickness of the plane of focus) than any ofthe individual source brightfield images.

As used herein, a “focal plane” refers to a plane arranged perpendicularto an axis of an optical microscope's lens at which a biologicalspecimen is observable at optimal focus.

As used herein, a “defocusing distance” refers to a distance above orbelow the focal plane such that the biological specimen is observableout of focus.

As used herein, a “confluence mask” refers to a binary image in whichpixels are identified as belonging to the one or more cells in thebiological specimen such that pixels corresponding to the one or morecells are assigned a value of 1 and the remaining pixels correspondingto background are assigned a value of 0 or vice versa.

As used herein, a “cell image” refers to an image generated based on atleast two brightfield images obtained at different planes to enhancecell contrast relative to the background.

As used herein, a “seed mask” refers to an image having a binarypixelation generated based on a set pixel intensity threshold.

As used herein, a “cell-by-cell segmentation mask” refers to an imagehaving binary pixelation (i.e., each pixel is assigned a value of 0 or 1by the processor) such that the cells of the biological specimen 110 areeach displayed as a distinct region-of-interest. The cell-by-cellsegmentation mask may advantageously permit label-free counting of cellsdisplayed therein, permit determination of the entire area of individualadherent cells, permit analysis based on cell texture metrics and cellshape descriptors, and/or permit detection of individual cellboundaries, including for adherent cells that tend to be formed insheets, where each cell may contact a number of other adjacent cells inthe biological specimen 110.

As used herein, “region-growing iteration” refers to a single step in aniterative image segmentation method by which regions-of-interest(“ROls”) are defined by taking one or more initially identifiedindividual or sets of pixels (i.e., “seeds”) and iteratively expandingthat seed by adding neighboring pixels to the set. The processorutilizes similarity metrics to determine which pixels are added to thegrowing region and stopping criteria are defined for the processor todetermine when the region growing is complete.

As used herein, a “trained model” refers to a model for predictionand/or classification (e.g., an artificial neural network, a Bayesianpredictor, a decision tree) whose parameters (e.g., weights, filter bankcoefficients), structure (e.g., number of hidden layers and/or units,pattern of interconnection of such units), or other properties ofconfiguration have been trained (e.g., by reinforcement learning, bygradient descent, by analytically determining maximum likelihood valuesof model parameters), based on a set of training data, to generate anoutput that is predictive for the class membership of a cell (e.g.,alive/dead, differentiated/undifferentiated).

Referring now to FIGS. 3-5, a method 300 is illustrated using thecomputing device of FIGS. 1-2. Method 300 includes, at block 305, aprocessor 202 generating at least one phase contrast image 400 of abiological specimen 110 comprising one or more cells centered around afocal plane for the biological specimen 110. Then, at block 310, theprocessor 202 generates a confluence mask 410 in the form of a binaryimage based on the at least one phase contrast image 400. Next, at block315, the processor 202 receives a first brightfield image 415 of one ormore cells in the biological specimen 110 at a defocusing distance abovethe focal plane and a second brightfield image 420 of the one or morecells in the biological specimen 110 at the defocusing distance belowthe focal plane. The processor 202 then generates a cell image 425 ofthe one or more cells in the biological specimen based on the firstbrightfield image 415 and the second brightfield image 420, at block320. At block 325, the processor 202 generates a seed mask 430 based onthe cell image 425 and the at least one phase contrast image 400. Andthe processor 202 generates an image of the one or more cells in thebiological specimen showing a cell-by-cell segmentation mask 435 basedon the seed mask 430 and the confluence mask 410, at block 330.

As shown in FIG. 3, at block 305, the processor 202 generating at leastone phase contrast image 400 of the biological specimen 110 comprisingone or more cells centered around the focal plane for the biologicalspecimen 110 includes the processor 202 both receiving a Z-sweep ofbrightfield images and then generating the at least one phase contrastimage 400 based on the Z-sweep of brightfield images. In variousembodiments, the biological specimen 110 may be dispersed within aplurality of wells in a well plate representing an experimental set.

In one optional embodiment, method 100 includes the processor 202 bothreceiving at least one fluorescent image and then calculating afluorescent intensity of the one or more cells in the biologicalspecimen 110 within the cell-by-cell segmentation mask 435. In thisembodiment, the fluorescent intensity corresponds to the level of aprotein of interest, e.g. antibodies that label a cell surface markerlike CD20 or an annexin-V reagent that induces fluorescencecorresponding to cell death. In addition, determining fluorescentintensity within individual cell boundaries may increase subpopulationidentification and permit calculation of subpopulation-specific metrics(e.g., an average area and eccentricity of all dying cells, as definedby the presence of annexin-V).

In another embodiment, at block 310, the processor 202 generating theconfluence mask 410 in the form of the binary image based on the atleast one phase contrast image 400 includes the processor 202 applyingone or more of a local texture filter or a brightness filter to enableidentification of pixels belonging to the one or more cells in thebiological specimen 110. Example filters can include, but are notlimited to local range filters, local entropy filters, local standarddeviation filters, local brightness filters and Gabor wavelet filters.Example confluence masks 410, are shown in FIGS. 4 and 5.

In another optional embodiment, the optical microscope 105 determinesthe focal plane of the biological specimen 110. In addition, in variousembodiments, the defocusing distance may range from 20 pm to 60 p.m. Theoptimal defocusing distance is determined based on the opticalproperties of the objective used, including the magnification andworking distance of the objective.

In a further embodiment shown in FIG. 5, at block 320, the processor 202generating the cell image 425 based on the first brightfield image 415and the second brightfield image 420 includes the processor 202enhancing the first brightfield image 415 and the second brightfieldimage 420 based on a third brightfield image 405 that is centered aroundthe focal plane utilizing at least one of a plurality of pixel-wisemathematical operations or feature detection. One example of apixel-wise mathematical operation includes addition, subtraction,multiplication, division or any combination of these operations. Then,the processor 202 calculates transform parameters to align the firstbrightfield image 415 and the second brightfield image 420 with the atleast one phase contrast image 400. Next, the processor 202 combinesbrightness levels for each pixel of the aligned second brightfield image420 by a brightness level of corresponding pixels in the aligned firstbrightfield image 415 to thereby form the cell image 425. Thecombination of brightness levels for each pixel can be achieved via anyof the mathematical operations described above. The technical effect ofgenerating the cell image 425 is to remove brightfield artefacts (e.g.,shadows) and enhance image contrast to increase cell detection for theseed mask 430.

In another optional embodiment, at block 320, the processor 202generating the cell image 425 of the one or more cells in the biologicalspecimen 110 based on the first brightfield image 415 and the secondbrightfield image 420 includes the processor 202 receiving one or moreuser-defined parameters that determine one or more threshold levels andone or more filter sizes. The processor 202 then applies one or moresmoothing filters to the cell image 425 based on the one or moreuser-defined parameters. The technical effect of the smoothing filtersis to further increase accuracy of cell detection in the seed mask 430and increase the likelihood that one seed will be assigned per cell.Smoothing filter parameters are chosen to adapt to different adherentcell morphologies, for example, flat versus rounded shape, protrusivecells, clustered cells, etc.

In a further optional embodiment, at block 325, the processor 202generating the seed mask 430 based on the cell image 425 and the atleast one phase contrast image 400 includes the processor 202 modifyingthe cell image 425 such that each pixel at or above a threshold pixelintensity is identified as a cell seed pixel, thereby resulting in theseed mask 430 having a binary pixelation. The technical effect of theseed mask's binary pixelation is to permit comparison with thecorresponding binary pixelation of the confluence mask. The seed mask'sbinary pixelation is also utilized as a starting point for theregion-growing iteration discussed below. For example, in yet anotheroptional embodiment, the seed mask 430 may have a plurality of seedsthat each correspond to a single cell in the biological specimen 110. Inthis embodiment, method 300 further includes, prior to the processor 202generating the image of the one or more cells in the biological specimenshowing the cell-by-cell segmentation mask 435, the processor 202comparing the seed mask 430 and the confluence mask 410 and eliminatingone or more regions from the seed mask 430 that are not arranged in anarea of the confluence mask 410 and eliminating one or more regions fromthe confluence mask 410 that do not contain one of the plurality ofseeds of the seed mask 430. The technical effect of these eliminatedregions is to exclude small bright objects (e.g., cell debris) thatgenerate a seed and to increase identification of seeds utilized in theregion-growing iteration described below.

In a further optional embodiment, at block 330, the processor 202generating the image of the one or more cells in the biological specimen110 showing the cell-by-cell segmentation mask 435 based on the seedmask 430 and the confluence mask 410 includes the processor 202performing a region-growing iteration for each of an active set ofseeds. The processor 202 then repeats the region-growing iteration foreach seed in the active set of seeds until a growing region for a givenseed reaches one or more borders of the confluence mask 410 or overlapswith a growing region of another seed. The active set of seeds isselected by the processor 202 for each iteration based on properties ofthe corresponding pixels' values in the cell image. In addition, thetechnical effect of using at least one phase contrast image 400, as wellas brightfield images 415, 420, 405, is that the seeds correspond toboth a bright spot in the cell image 425 and also areas of high texturein the phase contrast image 400 (i.e., overlap of the confluence mask410 with the seed mask 430 described in more detail below). Anothertechnical effect that results from using the confluence mask 410, the atleast one phase contrast image, as well as brightfield images 415, 420,405, is increased accuracy in the identification of individual celllocations and cell boundaries in the cell-by-cell segmentation mask 435that advantageously permits quantifying features like cell surfaceprotein expression, as one example.

In still another optional embodiment, method 300 may include theprocessor 202 applying one or more filters in response to user input toremove objects based on one or more cell texture metrics and cell shapedescriptors. The processor 202 then modifies the image of the biologicalspecimen showing the cell-by-cell segmentation mask in response toapplication of the one or more filters. Example cell texture metrics andcell shape descriptors include, but are not limited to, a cell's size,perimeter, eccentricity, fluorescent intensity, aspect ratio, solidity,Feret's diameter, phase contrast entropy and phase contrast standarddeviation.

In a further optional embodiment, the method 300 may include theprocessor 202 determining a cell count for the biological specimen 110based on the image of the one or more cells in the biological specimen110 showing the cell-by-cell segmentation mask 435. The foregoing cellcount is advantageously permitted as a result of defined cell boundariesshown in the cell-by-cell segmentation mask 435, shown for example inFIG. 4. In one optional embodiment, the one or more cells in thebiological specimen 110 are one or more of adherent cells andnon-adherent cells. In a further embodiment, the adherent cells mayinclude one or more of various cancer cell lines, including human lungcarcinoma cells, fibrocarcinoma cells, breast cancer cells, ovariancancer cells, or human microvascular cell lines, including humanumbilical vein cells. In an optional embodiment, the processor 202performs the region-growing iteration in such a way that differentsmoothing filters are applied to non-adherent cells, including humanimmune cells like PMBCs and Jurkat cells, than are applied to adherentcells to improve approximation of cell boundaries.

As one example, a non-transitory computer-readable medium having storedthereon program instructions that upon execution by a processor 202,cause performance of a set of acts that include the processor 202generating at least one phase contrast image 400 of a biologicalspecimen 110 comprising one or more cells based on at least onebrightfield image 405 centered around a focal plane for the biologicalspecimen 110. The processor 202 then generates a confluence mask 410 inthe form of a binary image based on the at least one phase contrastimage 400. Next, the processor 202 receives a first brightfield image415 of one or more cells in the biological specimen 110 at a defocusingdistance above the focal plane and a second brightfield image 420 of theone or more cells in the biological specimen 110 at the defocusingdistance below the focal plane. The processor 202 then generates a cellimage 425 of the one or more cells based on the first brightfield image415 and the second brightfield image 420. The processor 202 alsogenerates a seed mask 430 based on the cell image 425 and the at leastone phase contrast image 400. And the processor 202 generates an imageof the one or more cells in the biological specimen 100 showing acell-by-cell segmentation mask 435 based on the seed mask 430 and theconfluence mask 410.

In one optional embodiment, the non-transitory computer-readable mediumfurther includes the processor 202 receiving at least one fluorescentimage and the processor 202 calculating a fluorescent intensity of theone or more cells in the biological specimen within the cell-by-cellsegmentation mask.

In another optional embodiment, the non-transitory computer-readablemedium further includes the processor 202 generating the seed mask 430based on the cell image 425 and the at least one phase contrast image400. And the non-transitory computer-readable medium further includesthe processor 202 modifying the cell image 410 such that each pixel ator above a threshold pixel intensity is identified as a cell seed pixel,thereby resulting in the seed mask 430 having a binary pixelation.

In a further optional embodiment, the seed mask 430 has a plurality ofseeds that each correspond to a single cell. And the non-transitorycomputer-readable medium further includes, prior to the processor 202generating the image of the one or more cells in the biological specimen110 showing the cell-by-cell segmentation mask 435, the processor 202comparing the seed mask 430 and the confluence mask 410 and eliminatingone or more regions from the seed mask 430 that are not arranged in anarea of the confluence mask 410 and eliminating one or more regions fromthe confluence mask 410 that do not contain one of the plurality ofseeds of the seed mask 430.

In yet another optional embodiment, the program instruction causing theprocessor 202 to generate the image of the one or more cells in thebiological specimen 110 showing the cell-by-cell segmentation mask 435based on the seed mask 430 and the confluence mask 410 includes theprocessor 202 performing a region-growing iteration for each of anactive set of seeds. Then, the non-transitory computer-readable mediumfurther includes the processor 202 repeating the region-growingiteration for each seed in the active set of seeds until a growingregion for a given seed reaches one or more borders of the confluencemask 410 or overlaps with a growing region of another seed.

The non-transitory computer-readable medium further includes theprocessor 202 applying one or more filters in response to user input toremove objects based on one or more cell texture metrics and cell shapedescriptors. And the processor 202 modifies the image of the biologicalspecimen 110 showing the cell-by-cell segmentation mask 435 in responseto application of the one or more filters.

Referring now to FIG. 8, an exemplary method 800 for classification ofcells is illustrated using the computing device of FIGS. 1-2. Method 800includes, at block 805, a processor (e.g., processor 202) obtaining aset of images of a plurality of biological samples, wherein the set ofimages includes at least one image of each sample of the plurality ofbiological samples. Then, at block 810, the processor obtains anindication of a first set of cells within the plurality of biologicalsamples and obtaining an indication of a second set of cells within theplurality of biological samples, wherein the first set of cells isassociated with a first condition and the second set of cells isassociated with a second condition. Next, at block 815, the processordetermines, based on the set of images, the indication of the first setof cells, and the indication of the second set of cells, a firstplurality of sets of metrics, wherein the first plurality of sets ofmetrics comprise a set of metrics for each cell of the first set ofcells and a set of metrics for each cell of the second set of cells. Atblock 820, the processor uses a supervised learning algorithm togenerate, based on the first plurality of sets of metrics, a model todistinguish between cells in the first set of cells and cells in thesecond set of cells, thereby generating a trained model. At block 825,the processor determines, based on the set of images, a second pluralityof sets of metrics, wherein the second plurality of sets of metricscomprise a set of metrics for each cell present in a target sample.Then, at block 830, the processor classifies a cell in the targetsample, wherein classifying the cell comprises applying the trainedmodel to the set of metrics for the cell. The method 800 could includeadditional steps or features.

Referring now to FIG. 9, another exemplary method 900 for classificationof cells is illustrated using the computing device of FIGS. 1-2. Method900 includes, at block 905, a processor (e.g., processor 202) obtainingthree or more images of a target sample, wherein the target samplecomprises one or more cells centered around a focal plane for the targetsample, wherein the three or more images include a phase contrast image,a first brightfield image, and a second brightfield image, wherein thefirst brightfield image represents an image of the target sample focusedat a first defocusing distance above the focal plane, and wherein thesecond brightfield image represents an image of the target samplefocused at a second defocusing distance below the focal plane. Then, atblock 910, the processor determines a cell image of the target samplebased on the first and second brightfield images. Next, at block 915,the processor determines a target segmentation map for the target samplebased on the cell image and the phase contrast image. At block 920, theprocessor determines, based on the two or more images of the targetsample and the target segmentation map, a set of metrics for each cellpresent in the target sample. Then at block 925, the processorclassifies a cell in the target sample, wherein classifying the cellcomprises applying the set of metrics of the cell to a trainedclassifier. The method 900 could include additional steps or features.

Referring now to FIG. 10, another exemplary method 1000 forclassification of cells is illustrated using the computing device ofFIGS. 1-2. Method 1000 includes, at block 1005, a processor (e.g.,processor 202) obtaining two or more images of a target sample, whereinthe target sample comprises one or more cells centered around a focalplane for the target sample, wherein the two or more images include aphase contrast image and one or more brightfield images, wherein the oneor more brightfield images includes at least one brightfield image thatrepresents an image of the target sample that is not focused at thefocal plane. Then, at block 1010, the processor determines, based on thetwo or more images, a set of metrics for each cell present in the targetsample. Next, at block 1015, the processor classifies a cell in thetarget sample by applying a trained model to the set of metrics for thecell. The method 1000 could include additional steps or features.

As discussed above, a non-transitory computer-readable medium havingstored thereon program instructions that upon execution by a processor202 may be utilized to cause performance of any of functions of theforegoing methods.

IV. Experimental Results

Example implementations permit cell health to be tracked insub-populations over time. For example, FIG. 6A shows experimentalresults of a cell-by-cell segmentation mask, generated according to anexample implementation, for a phase contrast image response at 24 hoursafter a time course of HT1080 fibrosarcoma apoptosis following acamptothecin (CPT, cytotoxic) treatment. Cell health was determined withmultiplexed readouts of Incucyte® NucLight Red (nuclear viabilitymarker) and non-perturbing Incucyte® Caspase 3/7 Green Reagent(apoptotic indicator). FIG. 6B shows cell subsets classified based onred and green fluorescence, according to the implementation of FIG. 6A,using Incucyte® Cell-by-Cell Analysis Software tools. FIG. 6C showsthere was a decrease in the red population after CPT treatmentindicating loss of viable cells, increasing red and green fluorescenceindicating early apoptosis, as well as increasing green fluorescenceafter 24 hours indicating late apoptosis, according to theimplementation of FIG. 6A. FIG. 6D shows concentration response timecourses of the early apoptotic population (percentage of total cellsexhibiting red and green fluorescence), according to the implementationof FIG. 6A. Values shown are the mean±SEM of 3 wells.

In another example, FIG. 6E shows experimental results of a cell-by-cellsegmentation mask, generated according to an example implementation, fora cell image response at 24 hours after a time course of HT1080fibrosarcoma apoptosis following a cyclohexamide (CHX, cytostatic)treatment. Cell health was determined with multiplexed readouts ofIncucyte® NucLight Red (nuclear viability marker) and non-perturbingIncucyte® Caspase 3/7 Green Reagent (apoptotic indicator). FIG. 6F showscell subsets classified based on red and green fluorescence, accordingto the implementation of FIG. 6E, using Incucyte® Cell-by-Cell AnalysisSoftware tools. FIG. 6G shows there was a lack of apoptosis but adecrease in cell counts after CHX treatment (data not shown), accordingto the implementation of FIG. 6E. FIG. 6H shows concentration responsetime courses of the early apoptotic population (percentage of totalcells exhibiting red and green fluorescence), according to theimplementation of FIG. 6E. Values shown are the mean±SEM of 3 wells.

FIG. 7A shows a cell-by cell segmentation mask imposed over a phasecontrast image for label-free cell counting of adherent cells usingcell-by-cell segmentation analysis, generated according to an exampleimplementation via Incucyte® software. Various densities of A549 Cellslabelled with NucLight Red reagent were analyzed with both thelabel-free cell-by-cell analysis and with the red nuclear count analysisto validate the label-free counting over time. FIG. 7B shows thecell-by-cell segmentation mask according to FIG. 7A without the phasecontrast image in the background. FIG. 7C shows a time course of phasecount and red count data across densities, according to theimplementation of FIG. 7A. FIG. 7D shows a correlation of count dataover 48 hours and demonstrates R2 value of 1 with a slope of 1,according to the implementation of FIG. 7A. This has been repeatedacross a range of cell types. Values shown are the mean±SEM of 4 wells.

V. Example Classification of Cells

Algorithmic classification of cells, based on images of samplescontaining the cells, can facilitate a variety of applications. This caninclude quantifying properties of the cells and/or cells samples,quantifying the response of the cell samples to applied experimentalconditions (e.g., the toxicity or effectiveness of a putative drug ortreatment), or assessing some other information about the samples.Classification of the cells facilitates such applications by allowingthe number of cells of each class within a sample to be determined. Suchclassifications may include two-class classifications or classificationinto more than two classes. In some examples of classifications, cellsmay be classified as alive or dead, as a stem cell or a mature cell, asan undifferentiated cell or as a differentiated cell, as a wildtype cellor a mutant cell, epithelial or mesenchymal, normal or morphologicallyaltered by an applied compound (e.g., altered by application of acytoskeleton-targeting treatment compound), or between two or moreadditional or alternative classifications. Cells may also be assignedmultiple classes, selected from respective multiple different enumeratedsets of classes. For example, a cell could be classified as alive (frompossible classes of ‘alive’ and ‘dead’) and as differentiated (frompossible classes of ‘differentiated’ and ‘undifferentiated’).

Embodiments described herein accomplish classification of a particularcell by determining a set of metrics for the cell. The set of metrics isdetermined from one or more microscopic images of the cell. Ofparticular utility in determining such metrics are one or more defocusedbrightfield images of the cell, or composite images determined therefromand/or in combination with phase contrast images of the cell. Forexample, one or more metrics for a cell could be determined from each ofa phase contrast image of the cell and a cell image (determined asdescribed above) of the cell. The determination of the set of metricsgenerally includes segmenting the image(s) in order to determine whatportion of the image(s) corresponds to the cell. The segmentation itselfis determined based on one or more of the images as described elsewhereherein. Further, the segmentation may be used to determine one or moreof the metrics (e.g., the size of the cell, one or more metrics relatedto the shape of the cell, etc.). The set of metrics is then applied to amodel in order to classify the cell.

FIG. 11 depicts an example cell-by-cell segmentation mask (bright lines)imposed over a phase contrast image 1100 of a biological sample thatincludes a number of cells, including an example cell 1110. Thecell-by-cell segmentation mask delineates the portion of the phasecontrast image 1100 that corresponds to the cell 1110; this is indicatedby the dark line 1150 that indicates the portion of the cell-by-cellsegmentation mask corresponding to the example cell 1110. The portion ofthe phase contrast image 1100 within the dark line 1150 can be used todetermine one or more metrics for the example cell 1110 (e.g.,texture-related metric(s), intensity-related metric(s)), as can theportion 1150 of the cell-by-cell segmentation mask that delineates theexample cell 1110 (e.g., size-related metric(s), shape-relatedmetric(s)).

The segmentation of one or more microscopic images of a biologicalsample to localize cells within that sample may be accomplished usingone or more of the methods described above. Additionally oralternatively, one or more microscopic images of the sample could beapplied to a convolutional neural network that has been trained togenerate such a segmentation map. This could include applying a phasecontrast image and a cell image of a sample.

The segmentation map can be used to determine a size metric for thecell. This can include using the segmentation map to determine an areaof the cell, a number of pixels of an image that are occupied by thecell, a percent of the pixels and/or area of an image that is occupiedby the cell, a length of a perimeter of the cell, a maximal Feretdiameter of the cell, or some other metric related to the size of thecell.

The segmentation map can also be used to determine one or more shapedescriptor metrics for the cell. Such shape descriptor metrics can adegree of circularity of the cell, a degree of roundness of a convexhull of the cell, or a proportion of the convex hull of the cell that isoccupied by the cell, the aspect ratio of a cell (i.e., the ratio of thecell's maximal length to its orthogonal axis), the geographical centroidof the cell, the intensity-weighted centroid of the cell or thedifference between those two centroids, or some other metric related tothe cell shape.

Additional metrics can include metrics related to the texture and/orintensity of the cell, as depicted in one or more microscopic images ofthe cell. Such microscopic images of the cell could include phasecontrast images, brightfield images, fluorescence images, or otherimages of the cell. The images could include composite images. Suchcomposite images could include a cell image generated, as describedabove, from two or more brightfield images focused at respectivedifferent planes relative to the cell contents of a biological sample.Another example composite image is a composite of a phase contrast imageand one or more brightfield images (e.g., a composite of a phasecontrast image and a cell image). Determining such a texture orintensity-based metrics can include determining the metric based onpixels of the image(s) that correspond to a particular cell according toa segmentation map.

Texture metrics may be determined from variation and/or texture acrossthe set of pixels that represents a cell. This can include calculatingone or more metrics on a neighborhood basis, e.g., for a given pixel, atexture value could be determined based on the set of pixels thatsurrounds the given pixel within a specified distance. Such neighborhoodtexture values could then be averaged across the pixels for a cell toresult in an overall texture value for the cell. Such texture values mayinclude a range value that is the difference between the maximal andminimal intensity values within the set of pixels, a variance orstandard deviation, an entropy, a contrast value that is a measure ofthe local variations present in the set of pixels, a homogeneity valuethat is the measure of uniformity in the set of pixels, and/or sometexture-based measurement(s).

Intensity-based metrics can include a mean brightness of the cell in animage, a standard deviation of the brightness of the cell in an image, aminimum of the brightness of the cell in an image, a maximum of thebrightness of the cell in an image, a brightness of a specifiedpercentile of pixels of the cells in an image, a kurtosis or skewnessmeasurement of the distribution of brightness values across the cell inan image, or some other metric based on the intensity, or the variationthereof, of the cell in one or more images.

Once a set of metrics has been determined for a particular cell, the setof metrics can be used to classify the cell. This can include applyingthe set of metrics to a trained model. Such a model could include one ormore of a principal components analysis, an independent componentsanalysis, a support vector machine, an artificial neural network, alookup table, a regression tree, an ensemble of regression trees, adecision tree, an ensemble of decision trees, a k-nearest neighborsanalysis, a Bayesian inference, or a logistic regression.

The output of the model could be a simple indication of the determinedclass of the cell whose set of metrics was applied to the model.Alternatively, the model could output one or more values that areindicative of the class of the cell. Such a value could then be comparedto a threshold in order to classify the cell. For example, if the modeloutput value is greater than a threshold the cell could be classified as‘alive,’ while if the model output value is less than the threshold thecell could be classified as ‘dead.’ The value of such a threshold couldbe determined by an algorithm, e.g., as part of a process of trainingthe model based on training data. Additionally or alternatively, thethreshold could be set by a user. For example, the user could adjust thethreshold based on visual feedback that indicates, within one or moremicroscopic images, the classification of cells in the image(s). Theuser could adjust the threshold after an initial threshold is generatedvia an algorithmic process.

FIGS. 12A and 12B illustrate an example of a substantially real-time orotherwise iterative process of a user adjusting the threshold value andreceiving visual feedback regarding the effects of the adjustment on theclassification of cells in a biological sample. FIG. 12A depictselements of an example user interface during a first period of time. Theexample user interface includes a first annotated image 1200 a (e.g., anannotated phase contrast image) of a biological sample. The firstannotated image 1200 a is annotated to indicate the cells in the sampleand to indicate the classification of the cells according to a firstvalue of a threshold. As shown in FIG. 12A, a first class of cells areindicated by red coloration and a second class of cells are indicated byblue coloration.

The threshold can then be updated by a user input to a second value.Such an input could include the user pressing a real or virtual buttonto increment or decrement the value of the threshold, the user operatinga keypad or other means to input a value for the threshold, the usermoving a slider or dial to adjust the value for the threshold, or theuser engaging in some other user input action to adjust the threshold tothe second value. The second value of the threshold is then applied tore-classify the cells in the sample. This re-classification is thenvisually provided to the user in the form of an updated second annotatedimage 1200 b of the biological sample, shown in FIG. 12B. The secondannotated image 1200 b is annotated to indicate the cells in the sampleand to indicate the classification of the cells according to the updatedsecond value of the threshold. The classification of some of the cellschanged with the adjustment of the threshold, and so the secondannotated image 1200 b reflects this change. Such an update processcould be performed a plurality of times. For example, the updatingprocess could be performed at a rate of once per 20 milliseconds or atsome other rate to approximate real-time updating of the cellclassifications as a result of the user adjusting the threshold value.

Models used to classify cells can be trained using supervised trainingmethods and a suitable training dataset. The training dataset includes aset of metrics determined for each cell in two or more groups oftraining cells. Each of the groups of training cells corresponds to arespective class or set of classes that the model can be trained todistinguish. The sets of metrics in the training dataset can bedetermined as described above, by determining the set of metrics for aparticular training cell in a particular group based on one or moremicroscopic images of the particular training cell.

In some examples, the training cells could be disposed within wells ofthe same multi-well sample plate that contains target cells to beclassified based on the training cells. This has the advantage oftraining the model on training cells that have been exposed to the sameor similar environmental or other conditions as the target cells withoutrequiring manual annotation of large numbers of individual cells.Alternatively, the training cells could be disposed in wells of a firstmulti-well sample plate and the target cells could be disposed inwell(s) of a second, different multi-cell sample plate. Such first andsecond multi-well sample plates could be incubated in the same incubatoror otherwise exposed to the same or similar environmental conditions.

The variety of image(s) and/or metrics used to train the model could bethe same as or could differ from the variety of image(s) and/or metricsapplied to the trained model to classify unknown cells. For example, afluorescent marker could be present in the biological sample(s) thatcontain the training cells, but could be absent from samples containingunknown target cells to be classified. This could allow for improvedtraining of the model while avoiding the complication or confoundingnature of adding the fluorescent marker to a target sample. Additionallyor alternatively, the fluorescent marker could be used to assigntraining cells into respective groups prior to training a model.

Training cells in the two (or more) groups of training cells could beidentified in a variety of ways. In some examples, the groups oftraining cells could be manually identified by a user. This couldinclude the user manually indicating individual cells for each of thetwo or more groups. Such an indication could be performed using a userinterface that depicts images of the cells within a biological sample,with or without the images having been segmented already. Additionallyor alternatively, the user could manually indicate whole wells of amulti-well sample plate as corresponding to respective classes fortraining. Any cells detected in a well indicated in such a manner wouldbe assigned to the corresponding class to train the model. The usercould indicate such wells based on knowledge about the conditions of thewells. For example, a particular well could include a substance thatinduces cell death and the user could then indicate such a well ascontaining cells belonging to the ‘dead’ class for training a model.Indicating groups of training cells in such a well-by-well manner hasthe advantage of requiring a relatively low amount of user time andeffort (e.g., relative to the user indicating individual cells fortraining).

FIG. 13 depicts elements of an example user interface 1300 that could beused by a user to indicate one or more wells of a multi-well sampleplate as corresponding to one of two or more classes that a model canthen be trained to distinguish. The user interface 1300 depicts therelative locations of the wells of the multi-well sample plate, witheach well represented by a respective square. Additional informationabout each well could be provided. Such additional information couldinclude information about the contents of the wells, conditions appliedto the wells, images of the contents of the wells, or some otherinformation. The user could then indicate sets of the wells ascorresponding to respective classes. As shown, a user has indicated afirst set of wells 1310 a as corresponding to a first class (e.g., an‘alive’ class) and a second set of wells 1310 b as corresponding to asecond class (e.g., a ‘dead’ class).

Note that indication of sets of cells (e.g., by indicating individualcells, by indicating whole wells of a multi-well sample plate, byindicating the cells in concert with an automated or semi-automatedmethod) can include indicating the cells at one or more specified pointsin time. For example, indicating a first set of cells could includeindicating a well at a first point in time (e.g., when all or most ofthe cells in the well are alive, to indicate a set of alive cells) andindicating a second set of cells could include indicating the same wellat a second point in time (e.g., when all or most of the cells in thewell are dead, to indicate a set of dead cells).

The indicated sets of cells, or the sets of metrics determinedtherefrom, can be filtered or otherwise modified prior to using theresulting training data to train a model. This could be done in order toreduce the time or number of iterations required to fit the data, toresult in a more accurate model without overfitting the training data,or to otherwise improve the trained model and/or the process of trainingthe model. Such filtering or other pre-processing steps could includesynthetically balancing the training sets of cells, subsampling thetraining sets of cells, and/or normalizing the values of the determinedmetrics (e.g., normalizing each determined metric such that thepopulation of values of the metric, across all cells in the trainingdata, occupied a standard range and/or comported with a specifieddistribution).

Additionally or alternatively, the groups of training cells could beidentified by an algorithm or otherwise automatically orsemi-automatically identified. This could include using the presence orabsence of a fluorescent marker to identify groups of training cells.This could include obtaining fluorescent images of biological samplesthat contain the fluorescent marker and, based on the fluorescentimages, identifying first and second groups of cells in the sampleaccording to whether the cells have a mean fluorescence intensitygreater or lesser, respectively, than a threshold level.

In another example, an unsupervised training process could be used toclassify cells in training images. This could include identifying two ormore clusters of cells within the training images. A user could thenmanually classify a limited number of cells as belonging to respectiveclasses selected from a set of two or more classes. These manuallyclassified cells could be cells that had already been clustered by theunsupervised training process or could be novel cells. The manualclassification could then be used to assign the clusters of cells toappropriate classes within the set of two or more classes. The manualclassification could be on a cell-by-cell basis, on a whole-well basis,or some other manner of manual classification of cells.

The description of different advantageous arrangements has beenpresented for purposes of illustration and description, and is notintended to be exhaustive or limited to the examples in the formdisclosed. Many modifications and variations will be apparent to thoseof ordinary skill in the art. Further, different advantageous examplesmay describe different advantages as compared to other advantageousexamples. The example or examples selected are chosen and described inorder to best explain the principles of the examples, the practicalapplication, and to enable others of ordinary skill in the art tounderstand the disclosure for various examples with variousmodifications as are suited to the particular use contemplated.

VI. Experimental Classification Results

Classification of cells is improved when using one or more metricsdetermined from cell images (i.e., composite images determined from twoor more defocused brightfield images) of the cells. FIGS. 14A and 14Bshow the accuracy of classification of cells as alive or dead across anumber of samples that were treated with camptothecin (a cytotoxiccompound capable of causing cell death, “CMP”) or an experimentalcontrol compound (“VEH”). FIG. 13A shows the classification based on aset of metrics determined from the cell-by-cell segmentation mask (e.g.,area, perimeter) of the samples and the phase contrast images (e.g.,phase contrast mean brightness) of the samples. FIG. 13B shows theclassification based on the above metrics as well as additional metricsdetermined from cell images (e.g., cell image mean brightness) of thesamples. The overall accuracy, across all of the cells represented inFIGS. 14A and 14B, increased from 0.82 to 0.94, with the F1 statisticincreasing from 0.89 to 0.96 (using the alive cells as the ‘positive’class).

FIGS. 15A and 15B show the effect of this improved accuracy ofclassification of cells as alive or dead on the determined cell deathrate in a number of samples as a function of time. FIG. 15A shows asample of determined cell death rates over time as determined from thecell-by-cell segmentation mask (e.g., area, perimeter) of the samplesand the phase contrast images (e.g., phase contrast mean brightness) ofthe samples. The red trace is the rate as determined by the trainedmodel, while the blue trace is the true rate. FIG. 15B shows a sample ofdetermined cell death rates over time as determined from a trained modelusing the above metrics as well as additional metrics determined fromcell images (e.g., cell image mean brightness) of the samples.

The classification methods described herein facilitate classification ofcells with an accuracy that approximates the accuracy offluorophore-based methods. This allows for accurate classificationwithout the expense, complexity, or experimental confounding effectsthat may be associated with the use of fluorescent labels. In anexperiment, A549 cells were treated with increasing concentrations ofthe cytotoxic compound camptothecin (0.1-10 μM) for 72 h in the presenceof Annexin V reagent. The cells were classed as Dead or Live based onfluorescence Annexin response (live cells=low fluorescence, deadcells=high fluorescence). The results of the Annexin V-basedclassification are shown in FIG. 16A. The metric-based methods describedherein were used to train a model using label-free features of dead (10μM, 72 h) and live (vehicle, 0-72 h) cells. This model was then appliedto class cells as Live or Dead in order to obtain a % dead cells whichwas comparable to that of the Annexin V response. The results of thislabel-free classification are shown in FIG. 16B. FIG. 16C shows anoverlay of the concentration response curves of % death at 72h ascalculated using Annexin V or label-free methods, showing that thepredicted response over the concentration range is comparable, and EC50values were similar (Annexin V EC₅₀=6.6 E-07; Label-free EC50=5.3 E-07M⁻¹).

We claim:
 1. A method for classification of cells, the methodcomprising: obtaining a set of images of a plurality of biologicalsamples, wherein the set of images includes at least one image of eachsample of the plurality of biological samples; obtaining an indicationof a first set of cells within the plurality of biological samples andobtaining an indication of a second set of cells within the plurality ofbiological samples, wherein the first set of cells is associated with afirst condition and the second set of cells is associated with a secondcondition; based on the set of images, the indication of the first setof cells, and the indication of the second set of cells, determining afirst plurality of sets of metrics, wherein the first plurality of setsof metrics comprise a set of metrics for each cell of the first set ofcells and a set of metrics for each cell of the second set of cells;based on the first plurality of sets of metrics, using a supervisedlearning algorithm to generate a model to distinguish between cells inthe first set of cells and cells in the second set of cells, therebygenerating a trained model; based on the set of images, determining asecond plurality of sets of metrics, wherein the second plurality ofsets of metrics comprise a set of metrics for each cell present in atarget sample; and classifying a cell in the target sample, whereinclassifying the cell comprises applying the trained model to the set ofmetrics for the cell.
 2. The method of claim 1, wherein applying thetrained model to the set of metrics for the cell comprises generating amodel output value based on the set of metrics of the cell, and whereinclassifying the cell additionally comprises comparing the model outputvalue to a threshold value.
 3. The method of claim 2, furthercomprising: displaying an annotated image of the target sample, whereinthe annotated image of the target sample includes an indication of thecell and of the classification of the cell; receiving a user inputindicative of an updated threshold value; re-classifying the cell bycomparing the model output value to the updated threshold value; anddisplaying an updated annotated image of the target sample, wherein theupdated annotated image of the target sample includes an indication ofthe cell and of the re-classification of the cell.
 4. The method ofclaim 1, wherein determining the set of metrics for the cell comprisesdetermining at least one of: a size metric, a shape descriptor metric, atexture metric, or an intensity-based metric.
 5. The method of claim 1,wherein the trained model includes at least one of a principalcomponents analysis, an independent components analysis, a supportvector machine, an artificial neural network, a lookup table, aregression tree, an ensemble of regression trees, a decision tree, anensemble of decision trees, a k-nearest neighbors analysis, a Bayesianinference, or a logistic regression.
 6. The method of claim 1, whereinthe target sample comprises one or more cells centered around a focalplane for the target sample, and wherein the images of the set of imagesthat depict the target sample include a phase contrast image and one ormore brightfield images, wherein the one or more brightfield imagesincludes at least one brightfield image that represents an image of thetarget sample that is not focused at the focal plane.
 7. The method ofclaim 6, wherein the one or more brightfield images include a firstbrightfield image and a second brightfield image, wherein the firstbrightfield image represents an image of the target sample focused at afirst defocusing distance above the focal plane, wherein the secondbrightfield image represents an image of the target sample focused at asecond defocusing distance below the focal plane, and wherein the methodfurther comprises: determining a cell image of the target sample basedon the first and second brightfield images, wherein determining the setof metrics for the cell comprises determining at least one metric basedon the cell image.
 8. The method of claim 1, wherein a fluorescentmarker is present in cells of the first set of cells and in cells of thesecond set of cells, and wherein the fluorescent marker is not presentin the target sample.
 9. The method of claim 1, wherein the first set ofcells and the second set of cells are all disposed within wells of afirst multi-well sample plate, and wherein the target sample is disposedwithin a well of a second multi-well sample plate.
 10. The method ofclaim 1, wherein the first set of cells, the second set of cells, andthe target sample are all disposed within wells of a multi-well sampleplate.
 11. The method of claim 10, further comprising: displaying anindication of the relative locations of wells of the multi-well sampleplate, wherein the first set of cells is present in a first set of wellsof the multi-well sample plate, wherein the second set of cells ispresent in a second set of wells of the multi-well sample plate, whereinobtaining the indication of the first set of cells and the indication ofthe second set of cells comprises, subsequent to displaying theindication of the relative location of wells of the multi-well sampleplate, receiving a user input indicative of the relative location of thefirst set of wells and the relative location of the second set of wellswithin the multi-well sample plate.
 12. The method of claim 1, furthercomprising: prior to generating the trained model, pre-processing thefirst plurality of sets of metrics by performing at least one of:normalizing at least one metric in the first plurality of sets ofmetrics, synthetically balancing the first plurality of sets of metricsbetween the set of metrics for each cell of the first set of cells andthe set of metrics for each cell of the second set of cells, andsub-sampling the first plurality of sets of metrics.
 13. The method ofclaim 1, wherein the first set of cells and the second set of cellscontain a fluorescent marker, wherein the set of images of the pluralityof biological samples comprises at least one fluorescent image depictingthe first set of cells and at least one fluorescent image depicting thesecond set of cells, wherein obtaining the indication of the first setof cells within the plurality of biological samples comprises using theat least one fluorescent image depicting the first set of cells toidentify the first set of cells, and wherein obtaining the indication ofthe second set of cells within the plurality of biological samplescomprises using the at least one fluorescent image depicting the secondset of cells to identify the second set of cells.
 14. The method ofclaim 1, wherein classifying the cell in the target comprises at leastone of classifying the cell as alive or dead, classifying the cell as astem cell or a mature cell, classifying the cell as epithelial ormesenchymal, or classifying the cell as an undifferentiated cell or adifferentiated cell.
 15. A method for classification of cells, themethod comprising: obtaining three or more images of a target sample,wherein the target sample comprises one or more cells centered around afocal plane for the target sample, wherein the three or more imagesinclude a phase contrast image, a first brightfield image, and a secondbrightfield image, wherein the first brightfield image represents animage of the target sample focused at a first defocusing distance abovethe focal plane, and wherein the second brightfield image represents animage of the target sample focused at a second defocusing distance belowthe focal plane; determining a cell image of the target sample based onthe first and second brightfield images; determining a targetsegmentation map for the target sample based on the cell image and thephase contrast image; based on the two or more images of the targetsample and the target segmentation map, determining a set of metrics foreach cell present in the target sample; and classifying a cell in thetarget sample, wherein classifying the cell comprises applying the setof metrics of the cell to a trained classifier.
 16. The method of claim15, wherein determining the set of metrics for the cell comprisesdetermining at least one of: a size metric, a shape descriptor metric, atexture metric, or an intensity-based metric.
 17. The method of claim 1,wherein determining the set of metrics of the cell comprises determiningat least one metric of the set of metrics of the cell based on the phasecontrast image.
 18. The method of claim 1, wherein determining thetarget segmentation map for the target sample based on the first andsecond brightfield images comprises applying at least the first andsecond brightfield images and the phase contrast image to aconvolutional neural network.
 19. A method for classification of cells,the method comprising: obtaining two or more images of a target sample,wherein the target sample comprises one or more cells centered around afocal plane for the target sample, wherein the two or more imagesinclude a phase contrast image and one or more brightfield images,wherein the one or more brightfield images includes at least onebrightfield image that represents an image of the target sample that isnot focused at the focal plane; based on the two or more images,determining a set of metrics for each cell present in the target sample;and classifying a cell in the target sample by applying a trained modelto the set of metrics for the cell.
 20. The method of claim 19, whereinthe two or more images of the target sample include a first brightfieldimage and a second brightfield image, wherein the first brightfieldimage represents an image of the target sample focused at a firstdefocusing distance above the focal plane, wherein the secondbrightfield image represents an image of the target sample focused at asecond defocusing distance below the focal plane, and wherein the methodfurther comprises: determining a cell image of the target sample basedon the first and second brightfield images, wherein determining the setof metrics for the cell comprises determining at least one metric basedon the cell image.
 21. A non-transitory computer-readable medium,configured to store at least computer-readable instructions that, whenexecuted by one or more processors of a computing device, causes thecomputing device to perform controller operations to perform the methodof any of claim
 1. 22. A system for assaying biological specimens, thesystem comprising: an optical microscope; a controller, wherein thecontroller comprises one or more processors; and a non-transitorycomputer-readable medium, configured to store at least computer-readableinstructions that, when executed by the controller, cause the controllerto perform controller operations to perform the method of claim 1.